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Abstract 

Research has documented the use of tools in computer-based learning environments as problematic, that is, 
learners do not use the tools and when they do, they tend to do it suboptimally. This study attempts to disentangle 
cause and effect of this suboptimal tool use for experienced learners. More specifically, learner variables 
(metacognitive and motivational) were related to the tool presentation (non-/embedded), interventions, type of 
tool use (quantitatively and qualitative) and learners’ performance. One hundred and seventeen graduate students 
were assigned to one of five conditions (embedded and non-embedded with explained tool functionality, 
embedded and non-embedded with non-explained tool functionality and one control condition) to study a 
hypertext using semi-structured concept maps as the tools. Findings are discussed with respect to experienced 
learners’ role on tool use and performance. Although no differences among conditions and performance were 
found, results reveal that the self-regulation skill of organization and the explained tool functionality affected 
time on tool negatively, while the self-regulation skill of elaboration and perceived tool usability showed a 
positive effect. Time on tool influenced performance positively. Quality influenced performance negatively. It is 
argued that some tools and interventions are unnecessary for experienced learners. 

Keywords: computer-based learning environments, goal orientation, self-efficacy, semi-structured concept maps, 
tool interventions, tools presentation, experienced learners, learning outcomes 

1. Introduction 

Learners are often offered an array of different (learning) tools in computer-based learning environments 
(CBLEs). These CBLEs often assume learners are good judges of their learning needs and will therefore use the 
offered tools according to the tools’ main aim, which is to help the learners learn better (Aleven, Stahl, Schworm, 
Fischer, & Wallace, 2003). However, research has indicated that the use of tools is problematic: The presence of 
tools does not guarantee that tools will be used and if tools happen to be used, this usage does not guarantee 
effective tool use (Elen & Clark, 2006). In this regard, research has already documented that learners tend to 
avoid the use of tools (Aleven et ah, 2003; Perkins, 1985) and when they decide to use the tools, learners tend to 
use them inadequately (Aleven et ah, 2003; Perkins, 1985). 

The use of tools has been classified as either quantitative or qualitative. While quantity of tool use focuses on 
amount of tool use (e.g., time spent on tool and frequency of tool use) (Clarebout, Horz, Schnotz, & Elen, 2010; 
Viau & Larivee, 1993), quality of tool use tries to analyze how learners include the tools in their learning process 
(Jiang & Elen, 2011; Zumbach, 2009). The purpose of research exploring different ways of tool usage in CBLEs 
is to analyze the effects tool use may have on learning outcomes (i.e., performance) (Aleven et ah, 2003; 
Clarebout & Elen, 2006). 

The question remains whether these claims are generalizable for experienced learners, who have already gone 
through a number of years of advanced higher education and for whom CBLEs are often made for (Lust, Juarez 
Collazo, Elen, & Clarebout, 2012). According to Aleven et ah (2003), tool use improves with age, which is 
directly related to learners’ improvement of metacognitive and motivational variables (see also: Perkins, 1985). 
Moreover, the different characteristics of the tool itself (Perkins, 1985) may impact the (mis)use of tools in 
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addition to the influence (instructional) interventions may also have. Therefore, it is questionable whether 
experienced learners need tools. Moreover, if tools are used, it is also questioned how tool use can be described 
in terms of quantity and quality as well as how tool use impacts performance. 

1.1 The Nature of the Experienced Learner and Tool Use 

Within Perkins’s (1985) framework to increase the probability of effective tool use, two main types of learner 
variables are emphasized. Perkins suggests that (a) not only the tool should be present and hence functional, but 
also (b) learners should be able to recognize the tools’ functionality and (c) learners should be motivated to use 
the tool(s). This means that (a) using the tools will have significant effects on learning, that (b) the learners’ 
ability to recognize the functionality of the tool(s) implies that learners do not only recognize tools’ functionality 
but also know the relationship between the tools and learning. This process entails metacognitive thinking 
(variables). Lastly, it implies that (c) the learners’ motivation that might encourage tool use implies motivational 
variables. Consequently, metacognitive, and motivational variables seem crucial to understand optimal tool use. 

Metacognitive variables are related to the knowledge learners have about relationships between tools and 
learning (Elen, Lowyck, & Proost, 1996). In this sense, metacognitive variables give an indication of learners’ 
(meta-) understanding of how using tools will help them enhance their own learning (Elen et ah, 1996). 
Self-regulated learning (skills) perceptions and (instructional) conceptions (Gerjets & Hesse, 2004; Lowyck, 
Elen, & Clarebout, 2004), are examples of metacognitive variables that have been emphasized in investigations 
on tool use (e.g., Aleven et ah, 2003; Hartley & Bendixen, 2001; Lowyck et ah, 2004). Evidence suggests that 
there is a relationship between the tool usage and learner’s self-regulated learning skills (Greene & Azevedo, 
2007). When tools are used, learners that are self-regulated obtain significant higher learning gains in 
comparison with low self-regulated learners (Lajoie & Azevedo, 2006; Pressley, Snyder, Levin, Murray, & 
Ghatala, 1987). Moreover, high self-regulators seem to use tools more qualitatively (Clarebout et ah, 2010). 

Learners’ conceptions and perceptions have also been described as a type of metacognitive variable (Elen & 
Clarebout, 2006). Perceptions could be learners’ interaction of their conceptual residue of experiences (i.e., 
conceptions of ideas and theories on tools and tool usage) with the learning environment (Lowyck et ah, 2004). 
Theoretically, it has been suggested that conceptions and perceptions determine whether or not or how tools will 
be used (Gerjets & Hesse, 2004; Winne, 1985). Based on this assumption, if the tools’ functionalities are not 
conceived or perceived correctly, then tool use is ineffective. However, empirically speaking, the role of 
conceptions and perceptions on tool use still remains scarce (Gerjets & Hesse, 2004; Lowyck et ah, 2004), and 
the existing literature has revealed no effects of conceptions and/or perceptions on tool use (e.g., Clarebout & 
Elen, 2009). In neighboring fields, the Technology Acceptance Model (TAM) (Davis, Bagozzi, & Warshaw, 1989) 
specifies two kinds of perceptions (perceived usefulness and perceived ease of use) that influence system use. 
The TAM, however, focuses on the adoption of a system such as text editors, e-mails, spreadsheets, Microsoft 
Windows, personal computing (Legris, Ingham, & Collerette, 2003). Based on the TAM perceptions, and in an 
attempt to specify perceptions for tool use in CBLEs, research has specified two kinds of perceptions for tool use 
in CBLEs (Collazo, Elen, & Clarebout, 2012), which we will focus on. These are, first of all, perceived tool 
functionality, that is, the degree to which a learner believes that using a certain tool would enhance their 
performance in order to reach a goal; secondly, perceived tool usability, that is, the degree to which a learner 
believes that using a certain tool would be usable and easy to use. 

However, even if learners have enough metacognitive skills that would hypothetically influence tool use positively, 
if they are not motivated to use the tools, tools will be unlikely used (Perkins, 1985; Winne, 1985; Zimmerman, 
2000). Motivation, however, is a broad term which derives from many associated constructs exploring different 
motivational aspects (Murphy & Alexander, 2000). Empirical and theoretical literature explores motivation in 
relation to tool use and performance in terms of goal orientation and self-efficacy in CBLEs (Aleven et ah, 2003; 
Murphy & Alexander, 2000; Newman, 1994; Zimmerman, 2000). 

Goal orientation is defined as the set of personal beliefs that reflect the reasons and intentions to engage and 
achieve a task (Dweck, 1986). Two main goal orientation types have been mostly emphasized which are mastery 
orientation (defined in intrapersonal terms) and performance orientation (defined in normative terms) (Dweck & 
Legget, 1988). These are each operationalized on a scale ranging from mastery and performance avoidance 
(negative) to mastery and performance approach (positive) (Elliot & McGregor, 2001). Mastery approach refers to 
the focus of individuals on developing competence, expanding knowledge, task completion and understanding, 
learning, mastery, solving problems, and developing new skills. Performance approach is exemplified by learners’ 
concerns about how well they perform and how others perceive their behavior. It focuses on personal ability, a 
normative social comparison with others, and a desire for public recognition of performance. Mastery avoidance 
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and performance avoidance, on the other hand, are conceived to be orientations with a negative valence. Mastery 
avoidance refers to the focus individuals have about striving to avoid misunderstanding or failing, avoid making 
mistakes or doing anything wrong or incorrectly from an intrapersonal perspective (Elliot & McGregor, 2001). 
Performance avoidance focuses on avoiding normative competence, refers to low competence expectancies, fear 
of failure and avoidance of failure (Elliot & Church, 1997; Elliot & McGregor, 2001). 

Research has revealed a negative relationship between goal orientation, specifically, mastery approach and 
quantity of tool use (time spent on tool) (Clarebout & Elen, 2009) and a positive relationship between performance 
goal orientation (approach and avoidance) on quantity of tool use (frequency) (Crippen, Biesinger, Muis, & Orgill, 

2009) . However, Huet, Escribe, Dupeyrat, and Sakdavong (2011) reported a significant negative correlation 
between performance goal orientation (approach and avoidance) and quantity of tool use (frequency of tool use), 
which means that learners with high performance approach and avoidance accessed the tools less frequently (Huet 
et al., 2011). 

Self-efficacy is the key element of the social cognitive theory (Bandura, 1977). It is defined as personal beliefs 
about having the means to organize or execute the courses of action to perform effectively (Bandura, 1997), which 
implies that self-efficacy influences how learners approach goals, tasks, and challenges (Bandura, 1997), in this 
case tools. Although studies have revealed that high self-efficacious learners increase the frequency to access tools 
(quantity of tool use) (Wu, Lowyck, Sercu, & Elen, 2011), evidence has also suggested that the quantity of tool use 
is negatively influenced by high self-efficacy levels (less frequent tool access) (Jiang & Elen, 2010). This in 
contrast to quality of tool use, which is positively influenced (Jiang & Elen, 2010). 

In summary, although highly skilled learners are found to use tools, it remains unclear what and how learner 
variables are precisely determining effective tool use, which would lead to performance increase. 

1.2 The Characteristics of Tools 

A second line of research focuses on variables that are related to the type of tool and the tool presentation (Schnotz 
& Heiss, 2009). This paper will focus on the effects of cognitive tools on tool use (contrary to information tools or 
scaffold tools (Lust et al., 2012)). Cognitive tools not only enhance, extend or augment thinking, they also help 
learners accomplish cognitive tasks (Perkins, 1985) and provide opportunities to apply knowledge in meaningful 
activities (Lajoie & Azevedo, 2006). For example, the tool for this study, namely a concept map, is believed to be 
a graphical tool that facilitates meaningful learning by externalizing and structuring knowledge through 
meaningful relationships between interconnecting concepts, and it helps organize and represent knowledge by 
means of relationships among concepts (Novak & Canas, 2008; Novak & Gowin, 1984). Based on the 
characteristics of concept maps, they could be considered “(knowledge organization) cognitive tools” (liyoshi & 
Hannafin, 1998). This is because they help learners simplify unnecessarily complex cognitive tasks (Cognitive 
Load Theory) and facilitate self-regulated organization (Metacognition Theory) (liyoshi & Hannafin, 1998). 

According to Ruiz-Primo’s (2004) directedness continuum, concept maps can range from high-directed to 
low-directed. High-directed concept maps provide students with the concepts, connecting lines, linking phrases, 
and the structure of the map. Low-directed concept maps allow learners to decide which, how, and how many 
concepts they will include in their maps. High-directed concept maps have been often addressed as semi-structured 
concept maps (Afamasaga-Fuata’i & McPhan, 2009; Rodriguez, 2006; Zeilik et al., 1997). Zeilik et al. (1997) 
have made extensive use of semi-structure concept maps with undergraduate students, which had positive effects 
on learning. This type of concept maps are similar to multiple-choice tests, with the difference that semi-structured 
concept maps measure connected and hierarchical understanding (Rodriguez, 2006). 

The idea of tool presentation mainly refers to whether the use of the tools is obligatory or optional (Schnotz & 
Heiss, 2009), which is also appointed as embedded (obligatory) or non-embedded (optional) (e.g., Clarebout & 
Elen, 2006; Clarebout et al., 2010). This means, that when tools are embedded, learners have no choice but to 
use them; whereas when tools are non-embedded, learners have the freedom to decide whether or not to use the 
tools (Clarebout & Elen, 2006; Clarebout et al., 2010). 

The tool presentation is related to the learner control debate (Lawless & Brown, 1997). One standpoint on the 
learning control debate supports the idea of giving the learners control over their learning, for example, in the 
component display theory (Merrill, 1980, 1983) and thus by non-embedding tools. Another point of view claims 
that learning is not effective in environments with little control over learning. Regarding results related to tools’ 
presentation, and in line with learner control literature, research indicates that learners with embedded tools 
perform more optimally (Martens, Poelmans, Daal, & Valcke, 1993), but having non-embedded tools may have a 
positive influence on the quality of tool use which in turn influences performance positively (Clarebout et al., 

2010 ) . 


28 




www.ccsenet.org/hes 


Higher Education Studies 


Vol. 4, No. 1; 2014 


Greene and Land (2000) illustrate the role of tool presentation and tool type on tool use in CBLEs. They 
analyzed undergraduate students’ use of non-embedded information tools (internet search engine machine) and 
embedded scaffold tools (guiding questions) while they worked on a project. The results pointed out that learners’ 
quality of tool use in the non-embedded information tools was low. That means that they had difficulty using the 
tools properly and their searches were too broad or irrelevant in many of the cases. With respect to the embedded 
scaffold tool, learners tended to omit questions and/or give superficial answers, resulting in suboptimal tool use. 

1.3 Influencing Tool Use 

A third line of research has explored the use of interventions that may affect tool use (Clarebout et al., 2010). 
The implementation of interventions has been made in order to increase tool use probabilities and influence 
optimal tool use without “forcing” the learner by making the tool functionality more discernible (Lowyck et al., 
2004). Based on the functionalities learners ascribe to tools, the tool use will be determined (Winne, 1985). 
Interventions are said to provide guidance during a learning task (Atkinson, 2002), and encourage the use of 
tools that provide information that the learner has not accessed (Lee & Lehman, 1993). Interventions can also 
specify the functionality of the tools as the benefits that can be obtained by using the tool(s) (Clarebout & Elen, 
2008). These interventions have been operationalized as pedagogical agents (Atkinson, 2002), instructional cues 
(Lee & Lehman, 1993) and advice (Clarebout & Elen, 2008), among others. Studies on interventions have 
revealed positive effects on quantity of tool use and performance (Atkinson, 2002; Lee & Lehman, 1993), but 
has also shown mixed effects (Clarebout & Elen, 2008). 

2. The Present Study 

The theoretical framework described above details the role of different variables that can influence tool use and 
performance. However, clear and concrete unequivocal research of these effects in a sole investigation has yet to 
be studied, especially in the case of experienced learners. 

Therefore, in a context of graduate learners using a cognitive tool (semi-structured concept maps), this paper, 
first of all, studies the role of perceptions (perceived tool functionality and perceived tool usability), 
self-regulated learning skills, goal orientation, and self-efficacy on quantity and quality of tool use. Secondly, it 
analyzes the effects of tool presentation (embedded and non-embedded tool) on the use of semi-structured 
concept maps. Third, it examines the influence of an instructional intervention operationalized as the explanation 
of tool functionality on quantity and quality of tool use. Fourth, this paper explores the possible direct effects of 
quantity and quality of tool use on performance. Finally and most importantly, in order to see if the tools 
contribute to performance, effects of conditions on performance were analyzed. Consequently the following 
research questions are addressed: 

(1) Do the different conditions contribute to learners’ performance? 

(2) Do quantity and quality of tool use influence performance? 

(3) What is the effect of perceptions, self-regulated learning, goal orientation and self-efficacy on quantity and 
quality of tool use? 

(4) What is the effect of tool presentation (embedded vs. non-embedded tools) and interventions 
(non-/explanation of tool) functionality on quantity and quality tool use? 

3. Methodology 

3.1 Participants 

There were a total of one hundred and seventeen Dutch speaking participants. All participants had a BA degree 
(with at least three years of learning experience at the university level), and were currently enrolled in the 
preparatory master program in Educational Studies from the University of Leuven in Belgium. They were on 
average 23 years old ( SD = 4.18) and eighty-two percent were female. Based on their average score of the 
pre-test (M = 5.34, SD = 1.36), learners were categorized as low prior knowledge learners in which no 
significant difference among the conditions was observed. As Table 1 confirms, participants were high-skilled 
and highly motivated learners. 

3.2 Instruments 

3.2.1 Learning Environment 

A text—in Dutch—on water scarcity titled Waarom water broodnodig is “Why is water essential” (Raes, Geerts, 
& Vanuytrecht, 2009) was selected and adapted in format for a Macromedia Director program. The text was 
comprised of 1,544 words and two figures and was divided into five sections. After each section, there was a 
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concept map. A concept map represents visual relationships in intertwined circles (Novak & Canas, 2008; Novak 
& Gowin, 1984). In this study, the concept maps were represented in the traditional hierarchical fashion, namely 
the more general concepts were displayed at the top and the less general concepts at the bottom (Novak & Canas, 
2008). The concept maps were semi-structured, which means that learners had to complete an unfinished concept 
map with concepts from the text (see Figure 1). There were five conditions (further information see Tables 1 and 
4). Two conditions had the explanation of the tool functionality (see Figure 2) one of which had embedded tools 
and the other one non-embedded tools. The explanation in Figure 2 reads as: “ Every ; concept map will bring up a 
part of your knowledge. By completing the concept map, you will be able to make a better connection between 
the information provided and your daily life. You knowledge will also become more meaningful and as a 
consequence the chances that you will do better in the test will be stronger Two other conditions did not have 
any explanation of the tool functionality, and as in the other conditions, one had embedded tools and the other 
non-embedded tools. In the non-embedded conditions students could press a button to access the tool (see Figure 
2). In the embedded conditions, on the other hand, participants did not have this button and by clicking on the 
button of next page, they directly accessed the tool. One last condition was the control condition, which received 
neither tools nor tool explanations. 


Table 1. Overview of conditions 


Condition 

Tool 

provided? 

Embedded 

tool? 

Explanation of tool 
functionality? 

Control Condition 

No 

N/A 

N/A 

Non-embedded, Explained Functionality 

Yes 

No 

Yes 

Embedded, Explained Functionality 

Yes 

Yes 

Yes 

Non-embedded, Non-explained 
functionality 

Yes 

No 

No 

Embedded, Non-explained functionality 
N/A: Non-applicable 

Yes 

Yes 

No 
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Figure 1. Tool employed in the form of concept maps 

Note. Concepts enclosed in boxes. The empty concepts in darkened boxes with number 1, 2 and 3 and question 
marks had to be filled in the darkened space below the concept map. This concept map is a translation from the 
original one version which was in Dutch 
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‘More crop per drop’ 


Pagina 5/10 


Om de waterschaarste aan te pakken kan men beroep doen op alternative watervoorraden zoals zoutwater of afvalwater. Omdat gewassen 
geen zout irrigatiewater verdragen : moet zout water eerst ontzilt worden. Dat vraagt ontzettend veel energie en is economisch niet haalbaar 
voor de landbouw. Het duurzaam gebruik van afvalwater als irrigatiewater op grote schaal wordt dan weer afgeremd door de hoge kosten voor 
waterzuivering en kwaliteitscontrole. Een betere oplossing om de waterschaarste aan te pakken, bestaat erin overtollige neerslag tijdens het 
regenseizoen op te slaan. Xu verdwijnt dat regenwater in bepaalde regions vaak via de rivieren naar de oceanen. Vandaag gebeurt er heel wat 
onderzoek naar hoe deze neerslag ‘geoogst kan worden. Eenvoudige ingrepen op het veldbeheer kunnen al helpen om wat extra water op te 
slaan in de wortelzone zodat gewassen korte droogteperiodes kunnen overbruggen. 

.Vs aitematieve oplossing voor het v aterp Elke begrippenkaart zal een deel van je 
het is een werk van lange adem en op korte tij<kennis in kaart brengen. 


s tolerante gewasvarieteiten. Maar 
t antwoord op de waterschaarste 


bestaat enn de gewas-waterproductmteit te ve Door de begrippenkaarten aan te VUllen, za|' ceren Een aangepast waterbeheer 


maakt het mogelijk om de gewasproducthiteit. . . , ... 

het onderzoek dat hierover loop! tauten de AfJ e beter ln staat zl J n om verbanden te 

leggen tussen deze informatie en het 
dagelijkse leven. Meteen wordt je kennis 
ook meer betekenisvol met als gevolg dat 
de kans groter wordt dat je goed presteert 
op de eindtoets. 


oordt bondig de doelstelling van 


Explained tool 


functionality 



[ Vorige pagina 


Access to tool - (Begrippenkaart2) 

[Volgende pagina] 


Figure 2. Example of non-embedded condition with explained tool functionality 
Note. The box in gray appeared after students clicked on “volgende pagina ” (next page). 
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3.2.2 Perceptions 

The questionnaire of perceived usefulness (Davis, 1989), system functionality (Pituch & Lee, 2006) and 
perceived functionality (Cho et al., 2009) were integrated into a single questionnaire to explore perceived 
functionality. This questionnaire has previously been used and shown high reliabilities; (above. 80, 
Juarez-Collazo et al., 2012) and has integrated the three questionnaires in order to have a broader view of 
perceived tool functionality. The questionnaire consisted of 15 questions. For perceived usability, the 
questionnaire on perceived ease of use (Davis, 1989) was employed. Both questionnaires were adapted to tool 
use and translated into Dutch by three different researchers using the translation/back translation method in order 
to avoid semantic problems (Behling & Law, 2000). In each questionnaire participants responded on a six-point 
Likert scale where one indicated totally disagree and six totally agree. 

3.2.3 Self-Regulating Skills 

To measure self-regulated learning skills, we used a selection of the LIST questionnaire (Wild & Schiefele, 1994) 
which was translated in Dutch and has been used in earlier research showing high reliabilities (e.g., Clarebout et 
al., 2010; Wild & Schiefele, 1994). The selection included six out of the eleven elements. The selection included 
elements that were applicable to individual study learning materials. Those elements related to discussing 
learning materials with others were discarded. As a result, the items from elements appointed as organization, 
elaboration, repetition, monitoring/planning, critical thinking and effort were included and measured on a 
six-point Likert scale (one, totally agree; six, totally disagree). Organization focuses on the activities that are 
performed to reorganize learning material to perform optimally (e.g., I make tables, figures or drawings to 
structure course content better). Elaboration is related to study activities that are appointed towards a deeper 
understanding of the learning material (e.g., I try to make connections with related topics). Critical thinking 
focuses on activities that deepen the understanding of the material through a critical analysis of statements and 
justification contexts (e.g.. What 1 leam, I examine critically). Repetition is related to the memorization of facts 
and rules by simply repeating (e.g., I memorize the text content by repeating). Monitoring/planning focuses on 
“planning” and “monitoring” that serve to control the current self-regulation learning processes (e.g.. Before I 
start to study my learning material, I think about how I can proceed most effectively). Lastly, effort covers the 
extent to which increased efforts will be actively used to achieve academic and learning goals (e.g., I don’t give 
up, even if the material is complex). 

3.2.4 Goal Orientation 

Goal orientation was measured by integrating two questionnaires of Elliot et al. (Elliot & Church, 1997; Elliot & 
McGregor, 2001). The initial questionnaire of Elliot and colleagues (Elliot & Church, 1997) only measured three 
dimensions of goal orientation (mastery approach, performance avoidance, and performance approach); the 
revised and employed questionnaire (Elliot & McGregor, 2001) incorporated mastery avoidance and mastery 
approach making a 2 x 2 framework of goal orientation (performance avoidance, performance approach, mastery 
avoidance and mastery approach). We employed the same translation procedure as in the perceptions 
questionnaires. A six-point Likert scale was employed where one indicated totally disagree and six totally agree. 

3.2.5 Self-Efficacy 

The questionnaire consisted of eight questions using elements from the Motivated Strategies for Learning 
Questionnaire (MSLQ) (Pintrich, Smith, Garcia, & McKeachie, 1991) and the Self- and Task-Perception 
Questionnaire (STPQ). This adaptation has previously been used and shown high Cronbach alpha’s reliabilities 
above .80 (e.g., Jiang & Elen, 2010; Juarez-Collazo, Lust, Elen, & Clarebout, 2011). As in the previous 
questionnaires, participants responded on a six-point Likert scale. 

3.2.6 Quantity of Tool Use 

Quantity of tool use was analyzed in two ways: the frequency learners accessed the tools, that is, the amount of 
clicks made on the link to access tools in non-embedded conditions only and the proportional time participants 
spent on the tool (all conditions). Both clicks and time spent on the tool were saved in log files. The time spent 
on the tool was recorded in seconds. 

3.2.7 Quality of Tool Use 

Quality of tool use was analyzed through the answers provided in the concept maps blank boxes. Each concept 
map (N= 5) had three boxes to be completed. In each box, learners had to give the correct/exact concept based 
on the text they were reading. The exact concept or synonym to the exact concept was awarded with one point. 
For example in Figure 1, the answer to the empty concept number 3 is “meat consumption”. Learners who 
answered “non-vegetarian diet”, for instance, were also awarded a point. Other answers, such as “diet”, for 


32 




www.ccsenet.org/hes 


Higher Education Studies 


Vol. 4, No. 1; 2014 


example, were given zero points. This kind of answers reflected that the learners did not read thoroughly, and 
thus did not include the tools in their learning process (low quality of tool use). If learners obtained the answers 
for each concept map correct, they could obtain a maximum of 15 points. The answers on the concept maps were 
recorded as text files that were retrieved and revised after the participant concluded the activity. 

3.2.8 Learning Outcomes 

A pre-test was conducted in order to see if learners differ regarding their prior knowledge. The test consisted of 
ten multiple-choice questions where learners could obtain a maximum of 10 points. Performance was measured 
through a post-test that participants answered after they finished the hypertext task. It consisted of 16 items, 
seven of which were multiple-choice items, three were fill-in-the-blank sentences and the last six items were true 
or false statements. Participants could obtain a point for every correct answer, thus they could obtain a maximum 
score of 16 in the post-test. 

4. Procedure 

The experiment was spread over two sessions. During the first session, two researchers distributed the 
questionnaires to all participants who were in an auditorium following their Learning and Instruction class. They 
filled out the questionnaires on perceptions, self-regulated learning, goal orientation and self-efficacy. 
Afterwards, participants had the possibility to enroll in one of the 14 sessions available. A maximum of 15 
participants could attend each session. The second session was the “computer” session. Herein participants 
entered the room and sat in front of a computer. Participants were told about the structure of the second session. 
The hypertext started on page three. When the participants concluded with the hypertext, they were given the 
performance tests. 

For data analysis, reliabilities for the different scales measuring the learner variables were conducted and 
descriptive statistics were performed in order to identify how participants are positioned on the different learner 
variables scales. A check using MANOVA was also performed to see if there were any differences among the 
different conditions (experimental and control) regarding the learner variables (including the prior knowledge 
pre-test). The post-test (performance) was also included as dependent variable in this analysis in order to answer 
question one. In addition, descriptive statistics were included. 

To explore the possible effects of tool use on performance (second research question), a simple regression was 
performed with both indicators of quantity of tool use (only time) and quality of tool use as independent 
variables and performance as dependent variable. Then, we conducted a second regression with frequency of tool 
use as independent variable and performance as dependent variable. The control condition was not considered 
for these analyses. Next, to observe the relationships among all aspects of tool use (time, frequency, and quality), 
a correlation analysis was conducted. 

To identify whether any learner variables influenced tool use (research question three), three different multiple 
regression analyses were conducted. One analysis used the learner variables as independent variables and 
quantity (time spent on tool) as dependent variable. A second analysis had quality of tool use as a dependent 
variable. The last regression—only with non-embedded conditions—was conducted with the same independent 
variables and quantity of tool use (frequency) as the dependent variable. In case any covariate was found, the 
regression model was run in two blocks, putting in the first block the covariates and in the second block the other 
predictor variables. 

For the last research question (What is the effect of tool presentation, i.e., embedded vs. non-embedded tools) 
and the intervention (i.e., explanation of tool functionality), a MANOVA with the experimental conditions as 
independent variable and quantity (only time spent on tool) and quality of tool use as dependent variables was 
run. Next, a separate ANOVA was done to analyze the effects of the experimental non-embedded conditions on 
frequency of tool use. 

5. Results 

5 .1 The Nature of the Experienced Learner 

Reliabilities, shown in Table 2, appeared overall appropriate. As a consequence all the scales were included in 
the analyses. The descriptive statistics (Table 2) indicate that the means for the different learner variables scales 
were between 3.09 and 4.79, pointing out that learners’ differences are not pronounced on one of the scales. The 
descriptive statistics per condition show similar results. This table can be observed in appendix A. Thus, learners 
seemed highly skilled as highly motivated. 
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Table 2. Descriptive statistics of all learner variables including reliability scales confirming strong learning skills 
and high motivation 



N 

M 

SD 

a 

Perceived Functionality 

117 

3.80 

.69 

.89 

Perceived Usability 

117 

3.78 

.77 

.82 

Self-efficacy 

117 

3.63 

.71 

.89 

Goal orientation: 

Performance Approach 

117 

3.09 

.97 

.89 

Master Approach 

117 

4.79 

.58 

.71 

Performance Avoidance 

117 

3.71 

.96 

.84 

Master Avoidance 

117 

4.34 

1.13 

.88 

Self-regulation: 

Organization 

117 

4.43 

.81 

.80 

Elaboration 

117 

4.30 

.78 

.86 

Repetition 

117 

4.48 

.92 

.71 

Metacognition 

117 

4.24 

.61 

.73 

Critical Thinking 

117 

3.30 

.78 

.82 

Effort 

117 

4.31 

.81 

.77 


The MANOVA analysis indicated that there was no significant difference among conditions regarding 
perceptions, goal orientations, self-efficacy, self-regulation, prior knowledge and performance (Wilks’ statistics 
A = 0.53, F(60,384) = 1.15, p = .22, partial if = .15). However, given that the results using Roy’s largest root 
were significant (© = 0.29, F(15,101) = 2. 00, p < .05, partial if = .23), we examined the separate follow-up 
ANOVAs. The ANOVAs suggested that in two subscales of self-regulated learning there was a significant 
difference among conditions. These scales were critical thinking (F(4,112) = 3.41, p < .05, partial // 2 = .11) and 
repetition (F(4,112) = 2.49, p < .05, partial if = .08). Consequently, in the further analyses (question 4) critical 
thinking and repetition were considered as covariates. 

5.2 Performance of the Learner 

5.2.1 Research Question 1: Do the Different Conditions Contribute to the Learners’ Performance? 

Table 3 illustrates the descriptive statistics with the number of participants per condition, the performance mean 
scores, the standard deviations, the representative percentages of the raw mean scores and the percentage relative 
standard deviation. The separate ANOVAs pointed out that there was no significant difference in prior 
knowledge (F(4,112) = 1.20, p = .32, partial if = .04) or in performance (F(4,112) = 1.38, p = .25, partial rf = .05) 
among conditions. These results confirm that learners were quite experienced. Even though the overall mean 
value of prior knowledge of the material was relatively low (M = 53.38), learners were able to increase their 
understanding of the domain up to a very high level (Avg. M= 85.78). 


Table 3. Descriptive statistics showing performance in each condition 


Condition 

N 

M 

SD 

Control Condition 

23 

13.35 

2.31 

Non-embedded, Explained Functionality 

24 

13.21 

1.74 

Embedded, Explained Functionality 

23 

13.91 

2.07 

Non-embedded, Non-explained functionality 

24 

14.29 

1.63 

Embedded, Non-explained functionality 

23 

13.87 

1.29 
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5.2.2 Research Question 2: Do Quantity and Quality of Tool Use Influence Performance? 

The regression analyses shown in Table 4 suggested that both aspects of quantity of tool use (time and frequency) 
influenced performance significantly. Time spent on tool had a positive effect on performance which means that 
the more time the learners spent on the tool, the better the results in the post-test. However, the frequency of tool 
use revealed a negative relationship with performance, which means that the more times the tool was accessed, 
the poorer the results in the post-test. Time spent on the tool accounted for 6.76% of the variance whereas 
frequency of tool use accounted for 12.96% of the variance. Quality of tool use, on the other hand, showed no 
significant effects on performance (it only accounted for 1.69% of the variance). For the first regression model, 
VIF values were well below 10, namely 1.01 and the average VIF was 1.01. 


Table 4. Regression with tool use and performance 


(1) Performance 

All conditions 




(2) Performance 

Non-embedded conditions 



B 

SEB 

P 

B SEB 

P 

(Constant) 

11.04 

1.26 


(Constant) 


Quantity (time on tools) 

.002 

.001 

.26* 

Quantity (frequency) 

-.36 

Quality tool use 

.13 

.10 

.13 




Note (1): R 2 = .08,/? < .05. 
Note (2): R 2 = .13,/? < .05. 
* p < .05 


The further correlation analysis among quantity (time and frequency) and quality of tool use showed a 
significant relationship; namely, quality of tool use and frequency of tool use were significantly correlated r 
= 36, p < .05 at two-tailed. 

5.3 What Influences Tool Use? 

5.3.1 Research Question 3: What Is the Effect of Perceptions, Self-Regulated Learning, Goal Orientation and 
Self-Efficacy on Quantity and Quality of Tool Use? 

Given that high-skilled learners scored high and their performance was influenced by quantity of tool use, it is 
important to see what learner variables specifically influenced this tool use. Table 5 shows the effects of the 
different learner variables on tool use. Hierarchical regression analysis (in two steps) was run in order to control 
for the subscales of critical thinking and repetition (covariates). In the first step, the covariates were examined 
alone to observe the isolated effects on these variables on quantity (time and frequency) and quality of tool use. 
The results show that critical thinking had positive significant effects on quality of tool use but significant 
negative influence on quantity of tool use, specifically, time spent on tools. Each of these effects explained 4.84% 
of the variance in tool use. 
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Table 5. Flierarchical regression with learner variables and tool use 


(1) Quality of tool use 

(2) Quantity of tool use: 
time spent on tools 

(3) Quantity of tool 
frequency on tools 

use: 

B 

SEB 

P 

B 

SEB 

P 

B 

SEB 

p 

Step 1 

(Constant) 

13.80 

1.51 


499.45 

176.32 


30.27 

9.66 


Critical thinking 

-.53 

.25 

-.22* 

63.24 

29.85 

.22* 

-.44 

1.72 

-.04 

Repetition 

-.10 

.23 

-.05 

-16.41 

27.07 

-.06 

-1.04 

1.54 

-.10 

Step 2 

(Constant) 

14.81 

2.45 


131.06 

275.37 


35.06 

17.75 


Critical thinking 

-.37 

.40 

-.15 

-12.94 

45.14 

-.05 

-.81 

3.07 

-.07 

Repetition 

-.08 

.29 

-.03 

-8.28 

33.00 

-.03 

-.28 

1.93 

-.03 

Organization 

-.31 

.29 

-.13 

-67.87 

32.59 

-.25* 

3.20 

2.27 

.31 

Elaboration 

.00 

.43 

.00 

96.78 

47.77 

.35* 

-1.28 

2.98 

-.10 

Monitoring/planning 

-.19 

.48 

-.06 

16.05 

53.77 

.04 

1.28 

3.39 

.09 

Effort 

.52 

.32 

.22 

13.31 

35.77 

.05 

3.80 

2.21 

.33 

Performance Approach 

-.05 

.23 

-.03 

-13.64 

26.18 

-.06 

-3.93 

2.41 

-.39 

Master Approach 

-.53 

.45 

-.16 

23.08 

50.07 

.06 

-4.41 

3.45 

-.31 

Performance Avoidance 

.11 

.29 

.06 

9.26 

32.68 

.04 

-2.44 

2.20 

-.29 

Master Avoidance 

.20 

.28 

.12 

21.08 

31.17 

.11 

2.81 

2.31 

.36 

Perceived Functionality 

-.19 

.41 

-.07 

-54.80 

46.15 

-.17 

-6.43 

3.80 

-.57 

Perceived Usability 

-.21 

.36 

-.09 

81.50 

39.99 

.30* 

1.51 

2.93 

.14 

Self-efficacy 

.33 

.32 

.13 

11.17 

35.84 

.04 

2.85 

3.19 

.24 


Note (1): R 2 =.Q5,p=A2 for Step 1. AR 2 = .10, p = .64 for Step 2. 

Note (2): R 2 = .06,/?=.06 for Step \.AR 2 = .16,p = .15 for Step 2. 

Note (3)-non-embedded conditions only-: R 2 = .01,/? = .78 for Step 1. AR 2 = .20,/? = .68 for Step 2. 

* p < .05 

After controlling for critical thinking and repetition in the second model, the significance effects of critical 
thinking disappeared in both quality and quantity of tool use (time) and further significant relationships were 
retrieved. The subscales of self-regulated learning, namely organization and elaboration, and perceived tool 
usability revealed a significant effect on quantity of tool use (time) (see Table 5). Organization had a negative 
influence on quantity of tool use (time), which accounted for 6.25% of the variance. This suggests that strong 
self-organizers do not need tools as much as other learners. Elaboration and perceived usability both had a 
positive effect on quantity of tool use (time). The effect of elaboration accounted for 12.25% of the variance 
whereas perceived usability accounted for 9% of the variance in the outcome. For our model, the tolerance was 
above 0.2, it ranged from .34 to .94. The VIF values were well below 10, ranging from 1.1 to 2.6, and the 
average VIF was 1.98. 

Specifically, the established effect of the self-regulated learning skill of elaboration suggests that the more 
learners were oriented towards a deeper understanding by means of study activities, in this case the use of tools, 
the more time they spent on tools. The finding on the role of perceived tool usability, on the other hand, indicates 
that the more learners believed that using a certain tool would be usable and easy to use, the more time they 
spent on it. 

Effects of the other subscales of self-regulated learning, perceived tool functionality and the motivational 
variables of goal orientation, self-efficacy and on quantity of tool use (time) could not be retrieved. As for the 
other aspects of tool use (frequency and quality of tool use), no learner variables showed a significant 
relationship either. A correlation table can be observed in Appendix B. 
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5.3.2 Research Question 4: What Is the Effect of Tool Presentation (Embedded vs. Non-Embedded Tools) and 
Interventions (Non-/Explanation of Tool Functionality) on Quantity and Quality Tool Use? 

To understand whether interventions and tool presentation had an impact on tool use, we performed a MANOVA 
analysis, which showed that there was a significant effect of tool presentation and interventions 
(non-/embeddedness and non-/explained tool functionality) on quantity (time) and quality of tool use using 
Wilks’ statistics (A = 0.75, F(6,176) = 4.59, p < .001, partial rf = .14). The separate ANOVAs on the outcome 
variables only revealed a significant effect of condition on quantity of tool use, specifically, time spent on the 
tool (F (3,89) = 8.94, p < .001, partial tj 2 = .23), but not on quality of tool use (F(3,89) = .90, p = .45, partial rf 
= .03). The Tukey post hoc tests revealed that the embedded condition without explained functionality spent 
significantly more time on the tools than the embedded and non-embedded conditions with explained 
functionality (ps < .005 and < .001, respectively). The non-embedded condition without explained functionality 
also spent significantly more time on tools than the conditions with explained functionality (embedded ps < .05 
and non-embedded ps < .001). This means that the tool intervention (explained functionality) played a more 
relevant role in the relationship on tool use than the tool presentation (non-/embedded) and conditions without 
explained functionality spent proportionally more time on the tools than conditions with explained functionality 
(see Figure 3). 

Finally, the separate ANOVA analysis that was conducted to examine the possible effects of condition (only 
non-embedded without explained functionality) on frequency of tool use showed no significant effects (F(l, 45) 
= .52, p = .47, partial rf = .01). 



Condition 

Figure 3. Graph depicting the differences among conditions and quantity of tool use (time) 


6. Discussion 

6.1 Learner and Tool Variables on Tool Use 

This study was conducted in a CBLE using semi-structured concept maps as tools and a population considered to 
be “experienced”, in this case graduate students. Within this context, we investigated the relationships between 
metacognitive (perceptions and self-regulated learning) and motivational (goal orientation and self-efficacy) 
learner variables, tool variables, (tool presentation: non-/embedded tool), interventions (non-/explained tool 
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functionality), on quantity and quality of tool use. Furthermore, this study also set out to determine the 
relationships between quantity and quality of tool use and performance, as well as, the influence of tools on 
performance. Our results pointed out that the group of participants was very homogeneous and showed little 
variation. The average results on each of the scales of the learner variables showed that learners gave themselves 
high scores. 

The findings suggest that there is a strong relationship between metacognitive variables and the time spent on the 
tools. Metacognitive variables have an indirect impact on performance as the time on tools showed a positive 
influence on learning outcomes. The fact that motivational variables did not contribute significantly to tool use 
may be an indicator that experienced learners do not need to be self-efficacious or goal oriented. Metacognitive 
variables may have either moderated or overpowered the effect of motivational variables, given that learners 
with sufficient self-regulatory skills have the ability to monitor their understanding and willingness to set goals 
(Hartley & Bendixen, 2001). 

Additionally, the effects of tool presentation were not as strong as previous evidence suggests (e.g., Clarebout et 
al., 2010). It was the interventions that had stronger effects on tool use. This effect was negative suggesting that 
learners in the conditions without interventions, that is, without the explained tool functionality used tools more 
by spending proportionally more time on them. Within the learner control research, effective performance results 
from the interaction of the learners’ internal processes (ability to select and sequence strategies) and the stimulus 
materials (in this case tools) (Merrill, 1980). The learners in this study showed through their metacognitive 
variables (specifically self-regulation skill of elaboration and perceived tool usability—which showed an effect 
on tool use) their capability to regulate their internal processes. Therefore, the explanation of tool functionality 
may not help experienced learners and may therefore discourage them from using the tools in a CBLE. Therefore, 
the explanation of tool functionality may not always result in more optimal tool use as previously suggested 
(Winne, 1985) and reported (e.g., Clarebout & Elen, 2009). Moreover, these experienced learners may also see 
the explanation of the functionality as directed to novice learners and hence avoided the explanation because 
they saw it as a threat to their learning (Rogers & Freiberg, 1994). 

6.2 Tool Use and Performance 

Quantity of tool use, namely time spent on the tool, showed a positive relationship with performance. This 
relation was also strong (12.96% variance) given the specific population we investigated. This result is in line 
with previous research (e.g., Clarebout et al., 2010; Viau & Larivee, 1993). Viau and Larivee (1993) found not 
only a positive effect of time spent on tool, but also of frequency of tool use in performance. Our results also 
suggest an indirect effect of perceived tool usability and elaboration on performance, while explained tool 
functionality and self-organization had a negative indirect influence on performance. 

The other aspect of quantity of tool use, namely frequency of tool use, showed a negative influence on 
performance. Similarly, Jiang and Elen (2011) found that frequent use of cognitive tools, namely questions, led 
to poor performance. It appeared that reading back and forth between questions and paragraphs prompted high 
frequency of tool access which affected performance negatively (Jiang & Elen, 2011). Possibly learners, in this 
study, encountered difficulties while searching answers (also by going back and forth) for the concept maps 
which similarly occurred in previous research (Cerdan, Vidal-Abarca, Martinez, Gilabert, & Gil, 2009). 

Moreover, although quality of tool use did not have a significant influence on performance as found in previous 
research (e.g., Clarebout et al., 2010; Jiang & Elen, 2011); a significant positive correlation between quality of 
tool use and frequency was found. This result suggests that the high frequency of tool use is closely related to the 
quality of tool use and together affect performance negatively. This finding also suggests that non-embedded 
tools may hamper performance indirectly. Possibly for these experienced learners, in order to obtain more 
learning gains, embedding tools and allowing enough time on the tools seems more relevant than non-embedding 
tools and other measures of tool use (in this case quality and frequency of tool use). 

6.3 Tool Effects on Performance 

No difference among experimental and control conditions was observed in relation to performance. These results 
are related to previous studies (Martens, Valcke, & Portier, 1997; Zumbach, 2009). Martens, Valcke and Portier 
saw no difference among conditions with different tool variables in a CBLE and conditions without a CBLE. 
Martens and colleagues implied that CBLEs were as effective as non-CBLEs. Zumbach obtained his results 
comparing two experimental conditions (with a mind mapping tool and a two-columned text argumentation tool) 
with a control condition. Even though he found no advantages or disadvantages of the use/not use of tools on 
performance, he suggested that intrinsic motivation was larger in the experimental groups. Our results tell us, 
first, that experienced students without tools, interventions, or different tool presentations can perform similarly 
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to students facing all the variables aforementioned. Second, they suggest that tool variables and interventions do 
not harm or help learning outcomes (performance). 

In order to maximize the positive effects of tools, interventions, or different tool presentations in CBLEs, one has 
to make sure that learners are put in the “right” condition (Martens et al., 1997). In our case, learners at this level 
should be trusted to have the wisdom of knowing how tools help. Patronizing or underestimating students—in 
this case through the explanation of the tool functionality—whether they have prior knowledge or not, does not 
make learning environments (computer or otherwise) any better. These results are limited by the fact that, as also 
one of the reviewers indicated, the only tool used here was a semi-structured concept map. Future studies should 
include more low-directed concept maps. It is possible that experienced learners can benefit more from these 
types of tools. 

7. Conclusion 

Overall, this paper has given account of the problem that surrounds the use of tools in CBLEs and adds to the 
literature of tool use in CBLEs. Instructional designers or teachers should be aware that interventions, in this 
case explained functionality, may not be needed with learners with certain educational experience. It seems that 
such interventions have a negative impact on the time experienced learners spent on tools. It also appears that the 
explanation of tool functionality moderates the influence of tool presentation on tool use. 

Another contribution to research in CBLE based on the results of this study focuses on the relationships between 
learner variables and tool use. Although motivational variables (goal orientation and self-efficacy) did not affect 
tool use, their effects could be moderated by the effects of metacognitive variables (self-regulated learning and 
perceptions) on tool use which role was significant. The relevance of self-regulated learning and perceptions was 
clearly supported by our findings in which perceived tool usability, self-regulated learning: elaboration and 
organization showed a significant relationship on quantity of tool use (time spent on tool), the latter a negative 
one. 

A third contribution of this paper is that whereas the time spent on the tool had a positive impact on performance, 
frequency of tool use had a negative impact on performance. The negative effect of frequency of tool use on 
performance also relates indirectly to quality of tool use given that it showed a positive correlation to frequency 
of tool use. This finding questions whether quality of tool use, frequency of tools use and indirectly 
non-embeddedness of tools harm a potentially useful tool by causing it to become dysfunctional if overused. One 
of the implications that emerge from this finding relate specifically to whether frequency of tool use and quality 
could be measurements that should be considered when dealing with experienced learners. Similarly, the 
relevance of experienced learners in CBLEs is clearly supported when we could observe that there was no 
difference in performance between experimental conditions and control condition. More importantly, our 
findings question whether implementing tools in homogeneous groups with more experienced learners is 
relevant. 

Finally, these concluding remarks suggest courses of action in designing CBLEs for experienced learners. While 
it is clear that tools should be made available for learners in CBLEs giving experienced learners control of their 
learning, and thus by not embedding tools and avoiding explanation of tools’ functionalities, is an important 
practical implication for CBLEs. 
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